{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Guided Project 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Learning Objective:**\n",
    "\n",
    "* Learn how to adapt the tfx template to an existing model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this guided project, we will use the `tfx template` tool to create a TFX pipeline around the covertype dataset.\n",
    "The goal is to adapt the template pipeline to make use of the model code for the covertype dataset we  already developed in\n",
    "the first part of this course. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 1. Environment setup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Envirnonment Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setup the your Kubeflow pipelines endopoint below the same way you did in guided project 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ENDPOINT = '' # Enter your ENDPOINT here."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH=%env PATH\n",
    "%env PATH={PATH}:/home/jupyter/.local/bin"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "shell_output=!gcloud config list --format 'value(core.project)' 2>/dev/null\n",
    "GOOGLE_CLOUD_PROJECT=shell_output[0]\n",
    "\n",
    "%env GOOGLE_CLOUD_PROJECT={GOOGLE_CLOUD_PROJECT}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Docker image name for the pipeline image.\n",
    "CUSTOM_TFX_IMAGE = 'gcr.io/' + GOOGLE_CLOUD_PROJECT + '/tfx-pipeline'\n",
    "CUSTOM_TFX_IMAGE"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `skaffold` tool setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "LOCAL_BIN=\"/home/jupyter/.local/bin\"\n",
    "SKAFFOLD_URI=\"https://storage.googleapis.com/skaffold/releases/latest/skaffold-linux-amd64\"\n",
    "\n",
    "test -d $LOCAL_BIN || mkdir -p $LOCAL_BIN\n",
    "\n",
    "which skaffold || (\n",
    "    curl -Lo skaffold $SKAFFOLD_URI &&\n",
    "    chmod +x skaffold               &&\n",
    "    mv skaffold $LOCAL_BIN\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Modify the `PATH` environment variable so that `skaffold` is available:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this point, you shoud see the `skaffold` tool with the command `which`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!which skaffold"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 2. Copy the predefined template to your project directory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step, we will create a working pipeline project directory and \n",
    "files by copying additional files from a predefined template.\n",
    "\n",
    "You may give your pipeline a different name by changing the PIPELINE_NAME below. \n",
    "\n",
    "This will also become the name of the project directory where your files will be put."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PIPELINE_NAME = \"guided_project_2\"\n",
    "PROJECT_DIR = os.path.join(os.path.expanduser(\".\"), PIPELINE_NAME)\n",
    "PROJECT_DIR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TFX includes the taxi template with the TFX python package. \n",
    "\n",
    "If you are planning to solve a point-wise prediction problem,\n",
    "including classification and regresssion, this template could be used as a starting point.\n",
    "\n",
    "The `tfx template copy` CLI command copies predefined template files into your project directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tfx template copy \\\n",
    "  --pipeline-name={PIPELINE_NAME} \\\n",
    "  --destination-path={PROJECT_DIR} \\\n",
    "  --model=taxi"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%cd {PROJECT_DIR}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3. Browse your copied source files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The TFX template provides basic scaffold files to build a pipeline, including Python source code,\n",
    "sample data, and Jupyter Notebooks to analyse the output of the pipeline. \n",
    "\n",
    "The `taxi` template uses the same Chicago Taxi dataset and ML model as \n",
    "the [Airflow Tutorial](https://www.tensorflow.org/tfx/tutorials/tfx/airflow_workshop).\n",
    "\n",
    "Here is brief introduction to each of the Python files:\n",
    "\n",
    "`pipeline` - This directory contains the definition of the pipeline\n",
    "* `configs.py` — defines common constants for pipeline runners\n",
    "* `pipeline.py` — defines TFX components and a pipeline\n",
    "\n",
    "`models` - This directory contains ML model definitions.\n",
    "* `features.py`, `features_test.py` — defines features for the model\n",
    "* `preprocessing.py`, `preprocessing_test.py` — defines preprocessing jobs using tf::Transform\n",
    "\n",
    "`models/estimator` - This directory contains an Estimator based model.\n",
    "* `constants.py` — defines constants of the model\n",
    "* `model.py`, `model_test.py` — defines DNN model using TF estimator\n",
    "\n",
    "`models/keras` - This directory contains a Keras based model.\n",
    "* `constants.py` — defines constants of the model\n",
    "* `model.py`, `model_test.py` — defines DNN model using Keras\n",
    "\n",
    "`beam_dag_runner.py`, `kubeflow_dag_runner.py` — define runners for each orchestration engine\n",
    "\n",
    "\n",
    "**Running the tests:**\n",
    "You might notice that there are some files with `_test.py` in their name. \n",
    "These are unit tests of the pipeline and it is recommended to add more unit \n",
    "tests as you implement your own pipelines. \n",
    "You can run unit tests by supplying the module name of test files with `-m` flag. \n",
    "You can usually get a module name by deleting `.py` extension and replacing `/` with `..`\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m models.features_test\n",
    "!python -m models.keras.model_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 4. Create the artifact store bucket"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** You probably already have completed this step in guided project 1, so you may\n",
    "may skip it if this is the case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Components in the TFX pipeline will generate outputs for each run as\n",
    "[ML Metadata Artifacts](https://www.tensorflow.org/tfx/guide/mlmd), and they need to be stored somewhere.\n",
    "You can use any storage which the KFP cluster can access, and for this example we\n",
    "will use Google Cloud Storage (GCS).\n",
    "\n",
    "Let us create this bucket if you haven't created it in guided project 1.\n",
    "Its name will be `<YOUR_PROJECT>-kubeflowpipelines-default`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "GCS_BUCKET_NAME = GOOGLE_CLOUD_PROJECT + '-kubeflowpipelines-default'\n",
    "GCS_BUCKET_NAME"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!gsutil ls gs://{GCS_BUCKET_NAME} | grep {GCS_BUCKET_NAME} || gsutil mb gs://{GCS_BUCKET_NAME}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 5. Change the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we need to have the TFX pipeline dataset point to the GCS bucket\n",
    "where our covertype data is located. For that\n",
    "\n",
    "1. open `kubeflow_dag_runner.py`\n",
    "\n",
    "1. Set the variable `DATA_PATH` to `gs://workshop-datasets/covertype/small` which contains the covertype dataset\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6. Change the pre-processing module"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At this step we want to reuse the `features.py` and `preprocessing.py` we already have for the covertype dataset. To do that, we will just copy these files over the template ones:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "FEATURE_PY  = '../../tfx_pipelines/02-pipeline/solutions/pipeline/features.py'\n",
    "PREPROC_PY = '../../tfx_pipelines/02-pipeline/solutions/pipeline/preprocessing.py'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp {FEATURE_PY} ./models/features.py\n",
    "!cp {PREPROC_PY} ./models/preprocessing.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now when you run the tests in the two cells below they should fail \n",
    "because they were written for the template taxi dataset. \n",
    "\n",
    "**Exercise:** Modify the tests `features_test.py` and `preprocessing_test.py` as well as possibly the original modules until the tests pass."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m models.features_test"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python -m models.preprocessing_test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 6. Change the model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similarly as for the pre-processing we want to reuse the model we develop for the covertype dataset,\n",
    "so we will simply copy it over the template model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "MODEL_PY  = '../../tfx_pipelines/02-pipeline/solutions/pipeline/model.py'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!cp {MODEL_PY} ./models/keras/model.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Exercise:** Move the constants defined in `model.py` into `constants.py` and import them from `model.py` to respect the template structure."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 7. Run the covertype TFX pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a TFX pipeline using the `tfx pipeline create` command.\n",
    "\n",
    "**Note:** When creating a pipeline for KFP, we need a container image which will \n",
    "be used to run our pipeline. And skaffold will build the image for us. Because `skaffold`\n",
    "pulls base images from the docker hub, it will take 5~10 minutes when we build\n",
    "the image for the first time, but it will take much less time from the second build."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tfx pipeline create  \\\n",
    "--pipeline-path=kubeflow_dag_runner.py \\\n",
    "--endpoint={ENDPOINT} \\\n",
    "--build-target-image={CUSTOM_TFX_IMAGE}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While creating a pipeline, `Dockerfile` and `build.yaml` will be generated to build a Docker image.\n",
    "\n",
    "Don't forget to add these files to the source control system (for example, git) along with other source files.\n",
    "\n",
    "A pipeline definition file for [argo](https://argoproj.github.io/argo/) will be generated, too. \n",
    "The name of this file is `${PIPELINE_NAME}.tar.gz.` \n",
    "For example, it will be `guided_project_2.tar.gz` if the name of your pipeline is `guided_project_1`. \n",
    "It is recommended NOT to include this pipeline definition file into source control, because it will be generated from other Python files and will be updated whenever you update the pipeline. For your convenience, this file is already listed in `.gitignore` which is generated automatically."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now start an execution run with the newly created pipeline using the `tfx run create` command.\n",
    "\n",
    "**Note:** You may see the following error `Error importing tfx_bsl_extension.coders.` Please ignore it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tfx run create --pipeline-name={PIPELINE_NAME} --endpoint={ENDPOINT}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Or, you can also run the pipeline in the KFP Dashboard. The new execution run will be listed \n",
    "under Experiments in the KFP Dashboard. \n",
    "Clicking into the experiment will allow you to monitor progress and visualize \n",
    "the artifacts created during the execution run.\n",
    "\n",
    "However, we recommend visiting the KFP Dashboard. You can access the KFP Dashboard from \n",
    "the Cloud AI Platform Pipelines menu in Google Cloud Console. Once you visit the dashboard, \n",
    "you will be able to find the pipeline, and access a wealth of information about the pipeline. \n",
    "For example, you can find your runs under the Experiments menu, and when you open your\n",
    "execution run under Experiments you can find all your artifacts from the pipeline under Artifacts menu."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:** If your pipeline run fails, you can see detailed logs for each TFX component in the Experiments tab in the KFP Dashboard."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One of the major sources of failure is permission related problems. \n",
    "Please make sure your KFP cluster has permissions to access Google Cloud APIs.\n",
    "This can be configured [when you create a KFP cluster in GCP](https://cloud.google.com/ai-platform/pipelines/docs/setting-up),\n",
    "or see [Troubleshooting document in GCP](https://cloud.google.com/ai-platform/pipelines/docs/troubleshooting)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 8. Add components for data validation."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step, you will add components for data validation including `StatisticsGen`, `SchemaGen`, and `ExampleValidator`.\n",
    "If you are interested in data validation, please see \n",
    "[Get started with Tensorflow Data Validation](https://www.tensorflow.org/tfx/data_validation/get_started)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Double-click to change directory to pipeline and double-click again to open** `pipeline.py`. \n",
    "Find and uncomment the 3 lines which add `StatisticsGen`, `SchemaGen`, and `ExampleValidator` to the pipeline.\n",
    "(Tip: search for comments containing TODO(step 5):). Make sure to save `pipeline.py` after you edit it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You now need to update the existing pipeline with modified pipeline definition. Use the `tfx pipeline update` command to update your pipeline, followed by the `tfx run create` command to create a new execution run of your updated pipeline."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Update the pipeline\n",
    "!tfx pipeline update \\\n",
    "--pipeline-path=kubeflow_dag_runner.py \\\n",
    "--endpoint={ENDPOINT}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can run the pipeline the same way.\n",
    "!tfx run create --pipeline-name {PIPELINE_NAME} --endpoint={ENDPOINT}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check pipeline outputs\n",
    "\n",
    "Visit the KFP dashboard to find pipeline outputs in the page for your pipeline run. Click the Experiments tab on the left, and All runs in the Experiments page. You should be able to find the latest run under the name of your pipeline.\n",
    "\n",
    "See link below to access the dashboard:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('https://' + ENDPOINT)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 9. Add components for training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this step, you will add components for training and model validation including `Transform`, `Trainer`, `ResolverNode`, `Evaluator`, and `Pusher`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Double-click to open** `pipeline.py`. Find and uncomment the 5 lines which add `Transform`, `Trainer`, `ResolverNode`, `Evaluator` and `Pusher` to the pipeline. (Tip: search for TODO(step 6):)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Hints:** \n",
    "\n",
    "* In `pipeline.py` make sure you turn the cache of by setting `enable_cache=False` for debugging purposes (otherwise components that have been previously run won't be).\n",
    "\n",
    "* In `pipeline.py`, you'll need to set `infer_feature_shape=False`, otherwise you'll run into sparse/dense tensor mismatch.\n",
    "\n",
    "* In `pipeline.py`, you'll need to set the label `big_tipper` from the default template to the right label for our dataset in \n",
    "\n",
    "```python\n",
    "tfma.EvalConfig(\n",
    "      model_specs=[tfma.ModelSpec(label_key='big_tipper')],\n",
    "```\n",
    "\n",
    "* In `configs.py`, adapt the values of the variables `TRAIN_NUM_STEPS` and `EVAL_NUM_STEPS` to match what we had in the original covertype pipeline."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you did before, you now need to update the existing pipeline with the modified pipeline definition. The instructions are the same as Step 5. Update the pipeline using `tfx pipeline update`, and create an execution run using `tfx run create`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Verify that the pipeline DAG has changed accordingly in the Kubeflow UI:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tfx pipeline update \\\n",
    "--pipeline-path=kubeflow_dag_runner.py \\\n",
    "--endpoint={ENDPOINT}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"https://\" + ENDPOINT)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tfx run create --pipeline-name {PIPELINE_NAME} --endpoint={ENDPOINT}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When this execution run finishes successfully, you have now created and run your first TFX pipeline in AI Platform Pipelines!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Step 10. Try Cloud AI Platform Training and Prediction with KFP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TFX interoperates with several managed GCP services, such as [Cloud AI Platform for Training and Prediction](https://cloud.google.com/ai-platform/). You can set your `Trainer` component to use Cloud AI Platform Training, a managed service for training ML models. Moreover, when your model is built and ready to be served, you can push your model to Cloud AI Platform Prediction for serving. In this step, we will set our `Trainer` and `Pusher` component to use Cloud AI Platform services."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before editing files, you might first have to enable AI Platform Training & Prediction API.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Double-click pipeline to change directory, and double-click to open** `configs.py`. Uncomment the definition of `GOOGLE_CLOUD_REGION`, `GCP_AI_PLATFORM_TRAINING_ARGS` and `GCP_AI_PLATFORM_SERVING_ARGS`. We will use our custom built container image to train a model in Cloud AI Platform Training, so we should set `masterConfig.imageUri` in `GCP_AI_PLATFORM_TRAINING_ARGS` to the same value as `CUSTOM_TFX_IMAGE` above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Change directory one level up, and double-click to open** `kubeflow_dag_runner.py`. Uncomment `ai_platform_training_args` and `ai_platform_serving_args`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Update the pipeline and create an execution run as we did in step 5 and 6."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tfx pipeline update \\\n",
    "--pipeline-path=kubeflow_dag_runner.py \\\n",
    "--endpoint={ENDPOINT}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!tfx run create --pipeline-name {PIPELINE_NAME} --endpoint={ENDPOINT}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can find your training jobs in [Cloud AI Platform Jobs](https://console.cloud.google.com/ai-platform/jobs). If your pipeline completed successfully, you can find your model in [Cloud AI Platform Models](https://console.cloud.google.com/ai-platform/models)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## License\n",
    "\n",
    "<font size=-1>Licensed under the Apache License, Version 2.0 (the \\\"License\\\");\n",
    "you may not use this file except in compliance with the License.\n",
    "You may obtain a copy of the License at [https://www.apache.org/licenses/LICENSE-2.0](https://www.apache.org/licenses/LICENSE-2.0)\n",
    "\n",
    "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \\\"AS IS\\\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  See the License for the specific language governing permissions and limitations under the License.</font>"
   ]
  }
 ],
 "metadata": {
  "environment": {
   "name": "tf2-gpu.2-3.m71",
   "type": "gcloud",
   "uri": "gcr.io/deeplearning-platform-release/tf2-gpu.2-3:m71"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
